Putting the pieces together Cross-correlation as a unifying model of early multisensory integration

As a sea of fire fills the sky and voices of loved ones are drowned in a stream of explosions, New Year’s Eve can resemble a war zone. Remarkably, few perceive the chaotic event as disorienting. Instead of experiencing a disorganised flux of bursts and blazes, people link specific flashes and bangs – even if they were separated by hundreds of milliseconds – to a single cause, and enjoy a structured scene of fireworks. This process of ‘linking’ – technically, multisensory integration – may seem trivial but has remained scientifically obscure. How does the brain know which signals belong together? How does it integrate them? And how does it achieve this feat in an optimal fashion? In a new paper, Parise and Ernst [10] have made a significant contribution to the field by demonstrating how a single processing mechanism – correlation detection – might constitute a unifying principle for multisensory integration. The authors show how this mechanism – originally proposed to explain motion-detection [6, 13] – can faithfully describe human judgements in a multisensory temporal order and causality task. Moreover, equipped with the same, experimentally established parameters, their model could also reproduce a variety of earlier results, from synchrony judgements to Bayes-optimal cue integration. To put this finding into context, flash back to the 1950s, when the German physicist Werner Reichardt and biologist Bernard Hassenstein took up the problem of motion detection in the beetle Chlorophanus. These little creatures, the scientists observed, can reflexively follow a motion direction. But as a moving grid is shown to the beetle, each single photoreceptor only ‘sees’ alternations of dark and light. How does the beetle combine these alternations into a single motion direction? Hassenstein and Reichardt saw that two mirror-symmetric units could use cross-correlation to achieve directional tuning (see Fig 1a). If the sensors are excited sequentially, and one input signal is delayed while the other is not, the signals become synchronous for one, ‘preferred‘ direction but